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298 EDUCATIONAL AND PSYCHOLOGICAL RESEARCH PAPERS 
PUBLISHED IN 5 JOURNALS DURING 1966-67 WERE CLASSIFIED ON THE 
BASIS OF SPONSORSHIP, AUTHORS’ GENERAL INTENTIONS, DESIGN, 
ASSUMPTIONS, AND ANALYSIS. ADVANCES IN STATISTICAL THEORY 
(INCLUDING WORK ON GENERAL ANALYSIS OF VARIANCE DESIGNS, 
HYPOTHESIS TESTING IN FACTOR ANALYSIS, MULTIVARIATE ANALYSIS, 
FACTOR ANALYSIS, AND NON-METRIC DATA ANALYSIS) ARE DISCUSSED. 

BECAUSE OF THE RECENT PROLIFERATION OF STATISTICAL 
METHODOLOGY, RESEARCH IS SUGGESTED ON WHETHER DIFFERENT 
METHODS LEAD TO SUBSTANTIAL '.Y DIFFERENT INTERPRETATIONS OF 
DATA, AND WHETHER ALGORITHMS ARE COMPUTED DIFFERENTLY WHEN 
MACHINES AND PROGRAMS OF DIFFERENT DESIGNS ARE EMPLOYED. IN 
GENERAL, INCREASED USE OF MULTIVARIATE METHODS S 
RECOMMENDED. PAPER READ AT 1967 RESEARCH CONVOCATION OF THE 
EDUC. RES. ASSOC. OF N.Y. STATE (ALBANY, NOV. 1967). (AF) 






99ISI0Q3 



^ 001400 



Becent DeveXogpgimts in Educational Hesearcb Methodology* 



U.S. DEPARTMENT OF HEALTH, EDUCATION & WEIFARE 
OFFICE OF EDUCATION 



THIS DOCUMENT HAS BEEN KPSODIJCEO EXACTLY AS RECEIVED FROM THE 
PERSON OR ORGANIZATION ORIGINATING IT. POINTS OF VIEW OR OPiNiGiiS 
STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION 
POSITION OR POLICY. 



Rohert M* Pruzek, State University of New York at Albany 
David J. Kleixdce^ New York State Education Department 



#Japer Read on November 13, 196? at the I 967 Research Convocation of 
the Educational Research Association of New York State; in Albany, New York. 



Perhaps it is by new a truism that the most significant advances in any 
science are presaged by inventions of new machinery or methods for conductijag 
research. Thus, surely to a great extent, advances in education and psychology 
are dependent upon advances in research methodology. We have two related purposes 
in this short session. First, we wish to document the status of research methods 
which are typically used today and second, to point to ttew developments in 
methodological theory which appear likely to advance the practices of educational 
and psychological researchers in the future. 

The first part of this paper is devoted to a summary of the results of a 
survey of recently-published research in the educaticmal and psychological 
literature. In the second part we shall describe briefly certain new develqjpments 
in research methodology which seen to us to have special potential* Ois* most 
important oveirall objective is to encourage the reader to learn more about these 
new developments which are described in the references listed at the end of this 
paper. 

To begin, we surveyed 298 papers which were published during 1966 and 1967 
in five Journals which are identified in Table 1. 

Table 1. Journals Reviewed 



Journal 


Issues Covered 


No. of 






Papers 


American Educational Research Journal 


Jan. 1966-May 1967 


51* 


Educational and Psychological Measurement 


Spr. 1966-Sum. 1967 


66 


Journal of Educational Measurement 


Spr. 1966-S\am. 1967 


39 


Jounial of Educational Psychology 


Feb. 1966-Aag. 1967 


88it 


Multivariate Behavioral Research 


Jan. 1966-Jul. 1967 


5^ 



* Three papers fTom AERJ and two from JBP contained no e^licit or Inpliclt 
treatment of data. 

All papers were classified on the basis of sponsorship, authors' general 
intentions, design, assunptlons and analysis. Specifically, we classified each 



paper with respect to each of the five dichotomies:* 

1 A supported paper is one for which the cost of basic research was wholly 
or partially underwritten by an agency other than the one in which an 
author was regularly employed* All others are non~supported . 

2 A methodological paper is one whose discussion and conclusions indicate 
a desire to disseminate a method of analysis or to comment on an extant 
one* A substantive paper's discussion and conclusions focus on the 
individuals or attributes studied* 

3 An observational paper e3»mines intact grougas , applying the same tests 
and/or treatments to all individuals. A manipulative paper assigns 
different treatments* 

k A multivariate paper is one using some form of factor analysis » component 
analysis, discriminant function analysis, or multivariate analysis of 
varisuice* All others are univariate , including those using complex analysis 
of variance* 

5 A metric paper is one that at least implicitly assumes underlying measure- 
ment on an interval or ratio scale* All others are non-metric * 

Our concern here shall be with a review of salient features of the 
classification data which is presented in liable 2* 



*The reader will recognize that not all papers can easily be categorized along 
all of these variables* While some arbitrariness is inevitable and a re- 
examination might show sli^itly different results we feel that the dichoto- 
mizations presented herein are at least internally consistent* 
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Sable 2* Pive**way Classification of 298 Papers 
in Education and Psychology 

(^ote i Frequencies are listed in 2x2 tables belovf the diagonal ^ phi* co- 
efficients above.) 
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Consider first the support— non-support dichotony. From Table 2 we may see 
that granting agencies have come to be associated with a substantial proportion 
of published papers in this field. of the papers surveyed were prepared with 
at least some financial assistance from outside the institution in which the 
investigators worked. From the phi coefficient in row 1 and column 2 (.17) we 
see that) for these journals at least, supported work, more than non-suiported 
work, is likely to be of a substantive nature. It is interesting to note that 
there is so little correspondence between the support— non-support dimension and 
the multivai'iate dimension, despite the greater cost of data prtysessing of 
mu^.tivariate data. Perhaps this means that nearly everyone who currently 
publishes has a conrputer at his disposal for data analysis. 
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Consider next the metric-non-metric dichotomy. ?rm frequencies in Table 2, 

\ 

it may be seen that 83^ of the papers surveyed at least tacitly assume interval 
or ratio scales for measurement. Clearly this is not the appropriate place to 
debate relative virtues of parametric versus non-parametric procedures for data 
analysis in education in psychology; the point is that most researchers^ have 
bpted for parametric procedures in their analyses. 

There are several interesting findings vhich relate to the multivariate- 
univariate classification variable. First, 35^ of the papers examined involve 
some sort of multivariate procedure in either data analysis or discussion; 
this percentage drops to 22^ when we exclude Multivariate Bebaviaral Research , 
but still it is clear that roughly one fourth of the lapers currently reported 
cannot be fully understood unless the reader has some familiarity with smlti> 
variate methods. 

Vlhile it is not apparent from data in Table 2, it was clear from our 
examination of the papers, that very few studies using multivariate methods were 
concerned with cognitive variables; most such papers in these journals were 
concerned with personality— or affective— variables* Of course, many multivariate 
studies in the past have been concerned with achievement and aptitude variables, 
but it would be unfortunate if we were not to continue to have multivariate 
studies of cognitive variables. 

A finding which troubles us most and about which subsequent remarks will 
be particularly relevant is reflected by the high correlation (**»0) in Table 2 
between the univariate— multivariate dimension and the manipulative— observational 
dimension. When experimental methods, or treatment manipulations, are involved 
in a study one can be nearly certain that multivariate xoethods will not be 
eoployed in data analysis. See the frequencies in the associated four-fold table 
below the principal diagonal. Thus, it may be fairly stated that very little 
research of a scientific inferential nature is being conducted using multi- 




variate methods for analysis. Given that in education and psychology most of our 

t 

manipulative studies are concerned with more thsui one dependent variable, this 
seems to be particularly \mfortunate* 
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for now we move on to consider certain advances in methodological theory. 

Given our pessimism relating to the use of multivariate methods in research 
we now suggest some reasons for optimism, for persons who are likely to be 
gathering and analyzing multivariate data in the future we think that recent 
work on general analysis of variance designs by Bock and Bargmann suid work on 
hypothesis testing in factor analysis by Joreskog are espec3,ally worthy of 
attention. 

Bock and Bargmann published a quite readable paper (#2) on smalysis of 
convariance structures in which they describe methods for estimating variance 
coQponents in the following general situaticHEi: Siqppose one has gathered responses 

on a sanple of persons for a nvmiber of variables (such as tests, or test items) 
in which the variables can be placed in a fixed set of categories using a 
factorial or a hierarchical design. For example, classification variables mi^it 
be item format characteristics, content characteristics, features of the 
administration of the tests, etc* If an Investigator wishes to discover which 
classification variables or combinations of them are useful in discriminating 
among respondants, as well as the relative ’’importance" of these dimensions for 
distinguishing respondants, he may get this information directly from estixnates 
of variance components. !Tlie methods of Bock and Bargmann axe scale-free in the 
sense that units of measurement may differ across variables without influencing 
the results. While many applications of these methods can be imagined, we 
should be most interested to see studies in which achievement tests were asnranged 
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in a design with respect to various content characteristics, format character- 
istics, difficulty levels, etc., to learn more about the relative importance of 
such classifications in discriminating among students. Different types of 
students might also be studied. 

Karl Joreskog's work constitutes a large break-through in statistical methods 
for factor analysis. In a paper published in I966 and another presented in 
1967 Joreskog indicated his solutions to theoretical and conputational problems 
associated with hypothesis testing in maximum likelihood factor analysis. When 
an investigator has hypothesized (or fixed) values for any number of entries in 
a population factor pattern matrix, a factor intercorrelation matrix, or a 
matrix of uniqueness variances, Joreskog has now shown how it is possible to 
generate precise confidence intervals for each free (or unfixed) parameter. The 
method is scale free and very unrestrictive as to possible applications . Results 
reported in (9) shew that the widths of confidence intervals do not vary 
systematically with the size of the factor coefficients, or factor inter- 
correlations, but that they do vary substantially from one entry to another. 

Perhaps some persons tjIII be surprised to learn that by bridging the gap 
between Joreskog’s work and that of Bock and Bargmann it is possible to show 
substantial similarities between general mixed models in analysis of variance 
designs and statistical inferential factor analysis models. The major difference 
lies in the degree and type of specification required for design parameters. 
Perhaps we are fi n ally on the way to breakir^g down the esqperimentalist— 
correlationalist dichotomy which has so long been perceived as necessary by 
behavioral scientists. 

We refer you to two reviews in the Deceniber 1966 Review a£ Educational 
Research to document further, recent work in multivariate ane^ysis including 
multivariate analysis of variance and to point out numerous important advances 
in factor analysis methodology. Glass and Taylor (^) presented a paper which 


















i 



i 



i 


















I 



h 






i 



O 

-ERIC 












- 7 - 



was par”bic'ulf:j?ly readable and enlightened as to the significance of recent 
developments in factor analysis and Cramer and Bock (4) described recent 
developments in multivariate analysis , including multivariate analysis of 
variance* 



Other developments in research methodology which are generally not yet 
available in the literature are concerned with non-metric dats. analysis* For 
reasons which ^re probably obvious to most behavioral scientists^ loany Invest!- 
gators have begun to enpbasise approaebes to data. o.nalyais w^iich i»iko very weak 
assumptions about underlying metrics* Coonibs, in his 1964 book, A Theory of 
Data , was one of the first persons to make contributions in this area, but now 
we have recei'ii work especially by Guttman (6) which shows mors clearly the rele- 
vance of such approaches to conventional problems* These approaches emphasize 
identification of configurations or patterns in data as opposed to more common 
searches for underlying coordinate systems; computer programs have been worked 
and these developments will no doubt see important applications soon* 

A new model related to Guttman ’s work is that of Wiley (ll) on latent 
partition analysis. This model is appropriate for use when one has obtained 
several categorizations of a set of ob^jects or items and wishes to study relation- 
ships among the categorizations* For example, if one has obtained several 
mutually exclusive and exhaustive qualitative categorizations of test items with 
respect to some general sorting criterion, he may use this model to see whether 
the hypothesis of underlyliig latent categories is tenable. Enpirical work with 
this model has lead several of us to be optimistic about further applicr tlons* 

Let us close by suggesting some new directions for research, the need for 
which is a direct consequence of the ijroliferation of methodology. Surely it is 
becoming clear that for some types of dlata at least, there are several distinct 
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and theoi*etically compelling methods which are available for analysis# But 
little is known generally about whether different methods will lead to 
substantially different interpretations of data. Some veiy limited studies 
have already been desigiied to study "method variance" in relation to, say, 
"contejflfe variance" but it would be good to see intensive systematic studies in 
this direction. In addition to being generally ignorant about such differences 
in methods, we are also quite ignorant about differences in congputing algorithms 
•^ihen machines and programs of different designs are engployed. Obviously there 
will he no end to methodological problems of this kind but if we are to get on 
to the important substantive problems of educational and psychological research 
we will need first to obtain answers to these questions about differences in 



methods. 
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